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Abstract 


At  present,  the  System  Readiness  Level  (SRL),  as  developed  by  the  Systems 
Development  &  Maturity  Laboratory  (SysDML)  at  Stevens  Institute  of  Technology,  is  a 
descriptive  model  that  characterizes  the  effects  of  technology  and  integration  maturity 
on  a  system  engineering  effort  a  systems  development  program.  One  of  the  current 
deficiencies  in  system  maturity  assessments  (measure  of  readiness)  is  that  it  is 
performed  independent  of  any  systems  engineering  tools  or  supporting  artifacts,  which 
could  reduce  the  level  of  subjectivity  in  an  assessment  and  reliability  in  the  results.  The 
advent  of  system  engineering  modeling  tools  has  enabled  system  architects  to  better 
understand  a  system  by  depicting  various  views  of  the  system  and  its  components.  For 
this  purpose,  architectural  frameworks  have  been  introduced  for  various  domains  and 
industries  to  support  a  common  language  and  set  of  tools  for  developing  a  system.  One 
of  the  widely  adopted  frameworks  in  the  defense  sector  of  the  United  States  is  the 
Department  of  Defense  Architecture  Framework  (DoDAF).  In  addition,  Department  of 
Defense  (DoD)  subcontractors  have  adopted  DoDAF  as  part  of  their  systems 
engineering  process,  and  industry  consortia  are  currently  working  on  adopting  the 
DoDAF  vocabulary  and  products  to  complement  their  standardized  approaches  to 
systems  and  software  development.  With  the  current  challenges  in  systems  maturity 
assessment  and  the  advancement  of  systems  engineering  architecture  tools,  this 
research  has  attempted  to: 

•  Identify  the  systems  engineering  architectural  artifacts  that  support  the 
assessment  of  a  technology  maturity  (via  Technology  Readiness  Levels), 
integration  maturity  (via  Integration  Readiness  Levels),  and  likewise  system 
maturity  (via  System  Readiness  Levels); 

•  Correlate  systems  engineering  architectural  artifacts  to  supported  views  and 
artifacts  within  the  DoDAF  that  enable  TRL  and  IRL  assessment;  and 

•  Develop  a  maturity  assessment  tool  that  works  with  standard  industry  SE 
architecture  tools  (e.g.  Sparx  Enterprise  Architect,  IBM  Rhapsody). 
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1  Summary 


At  present,  the  System  Readiness  Level  (SRL),  as  developed  by  the  Systems 
Development  &  Maturity  Laboratory  (SysDML)  at  Stevens  Institute  of  Technology1 * * * * * 7,  is  a 
descriptive  method  that  characterizes  the  effects  of  technology  and  integration  maturity 
on  the  system  engineering  effort  of  a  Department  of  Defense  (DoD)  program.  The  SRL 
and  supporting  assessment  methodology  has  proven  itself  to  be  a  promising  mechanism 
for  understanding  the  effects  of  technology  and  integration  maturity  in  a  systems 
engineering  context.  In  addition,  the  current  tools  and  methods  have  demonstrated 
utility  for  defining  system  status  and  providing  leading  indicators  of  integration  risk. 
While  the  SRL  method  has  been  subjected  to  a  series  of  validations  with  DoD  programs 
and  organizations  (e.g.  US  Army  ARDEC,  NAVSEA  PMS  420,  Lockheed  Martin, 
Northrop  Grumman),  it  still  has  not  reduced  the  level  of  subjectivity  in  the  assessment 
and  reliability  in  the  results.  The  success  of  the  SRL’s  implementation  thus  far 
highlights  the  potential  benefits  of  extending  the  research  to  explore  the  application  of 
the  SRL  to  broader  areas  of  the  systems  engineering  and  management  domains, 
particularly  with  respect  to  systems  of  systems  implementations,  where  validated 
models  and  supporting  tools  are  lacking. 

One  of  the  current  deficiencies  in  system  maturity  assessments  is  that  it  is  performed 
independent  of  any  systems  engineering  tools  or  supporting  artifacts,  which  could 
reduce  the  level  of  subjectivity  in  an  assessment  and  reliability  in  the  results.  Within  the 
methods,  processes,  and  tools  of  systems  engineering  architecting,  there  exists  a 
substantial  base  of  architectural  artifacts  that  have  the  potential  to  significantly  reduce 
the  subjectivity  and  in  essence  increase  the  reliability  in  a  system  maturity  assessment. 

The  advent  of  system  engineering  modeling  tools  has  enabled  system  architects  to  better 
understand  a  system  by  depicting  various  views  of  the  system  and  its  components.  For 
this  purpose,  architectural  frameworks  have  been  introduced  for  various  domains  and 
industries  to  support  a  common  language  and  set  of  tools  for  developing  a  system. 
Architectural  frameworks  support  the  need  for  a  more  structured  approach  to  manage 
complexity  whilst  balancing  all  appropriate  user  perspectives.  One  of  the  widely  adopted 
frameworks  in  the  defense  sector  of  the  United  States  is  the  Department  of  Defense 
Architecture  Framework  (DoDAF).  In  addition,  DoD  subcontractors  have  adopted 
DoDAF  as  part  of  their  Systems  Engineering  process,  and  industry  consortia  are 
currently  working  on  adopting  the  DoDAF  vocabulary  and  products  to  complement  their 
standardized  approaches  to  systems  and  software  development.  Although  there  are  26 


1  For  a  detailed  description  of  the  SRL  methodology  see  Sauser,  B.,  J.E.  Ramirez-Marquez,  D.  Nowicki,  A. 

Deshmukh,  and  M.  Sarfaraz.  Development  of  Systems  Engineering  Maturity  Models  and  Management 

Tools.  Systems  Engineering  Research  Center  Final  Technical  Report  2011-TR-014,  January  2011 
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views  to  document  the  entire  architecture,  there  are  a  handful  of  views  that  can  be  used 
for  the  purpose  of  system  maturity  assessment. 

Thus  with  the  current  challenges  in  systems  maturity  assessment  and  the  advancement 
of  systems  engineering  architecture  tools,  this  research  seeks  to: 

[1]  Identity  the  systems  engineering  architectural  artifacts  that  support  the 
assessment  of  a  technology  maturity  (via  Technology  Readiness  Levels), 
integration  maturity  (via  Integration  Readiness  Levels),  and  likewise  system 
maturity  (via  System  Readiness  Levels); 

[2]  Correlate  systems  engineering  architectural  artifacts  to  supported  views  and 
artifacts  within  the  DoDAF  that  enable  TRL  and  IRL  assessment;  and 

[3]  Develop  a  maturity  assessment  tool  that  works  with  standard  industry  SE 
architecture  tools  (e.g.  Sparx  Enterprise  Architect). 
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2  Introduction 


Defense  programs  are  often  balancing  against  schedule  slippages,  cancellations,  and 
failure  to  meet  performance  objectives.  In  addition,  numerous  reports  have  described 
the  challenges  of  maturity  as  it  relates  to  integrating  technology  solutions  into  systems. 
To  that  end,  the  Technology  Readiness  Level  (TRL)  has  been  used  within  the 
Department  of  Defense  (DoD)  as  a  metric  in  assessing  the  risks  associated  with  a 
developing  or  acquired  technology  for  a  system  solution.  However,  one  of  the 
deficiencies  in  using  the  TRL  metric  is  that  estimates  of  maturity  can  be  reliant  on 
subjective  assessments  (Mahafza  2005;  Azizian  2009;  Sauser  and  Ramirez-Marquez 
2009;  Magnaye,  Sauser  et  al.  2010).  Although  there  are  guidelines  and  tools  to  support 
the  assessment  process  (Nolte,  Kennedy  et  al.  2003;  DoD  2009),  the  final  estimation  of 
maturity  is  left  to  the  evaluator(s)  (Tan,  Sauser  et  al.  2011). 

It  is  the  goal  of  this  research  to  lay  the  foundations  for  the  formulation  of  a  more 
informed  decision  support  framework  and  supporting  tools  that  will  assist  practitioners 
and  managers  in  measuring  and  determining  the  maturity  of  technology  and  their 
requisite  integrations.  To  accomplish  this,  maturity  artifacts,  the  information  needed  by 
decision  makers  to  make  informed  decisions,  are  identified  from  standardized  sources 
of  information  and  mapped  to  system  architectural  information  to  assist  in  maturity 
assessment. 

Architectures  facilitate  decision  making  by  conveying  the  necessary  information  to  the 
decision  maker  by  presenting  architecture  information,  and  the  TRL  and  IRL  provide  a 
metric  to  assess  the  maturity  of  a  technology  and  their  integrations  at  any  given  time. 
Architecture  data  supports  acquisition  program  management  and  systems  development 
by  representing  system  concepts,  design,  and  implementation  as  they  mature  over  time, 
which  enable  and  support  operational  requirements  (DoDAF  2007).  Therefore,  this 
research  explores  the  combined  use  of  the  Department  of  Defense  Architecture 
Framework  (DoDAF)  with  TRL  and  the  Integration  Maturity  Level  (IRL)  metrics  for 
maturity  assessment. 

The  development  of  this  research  is  intended  to  lead  to  a  more  informative  and  less 
subjective  method  for  the  assessment  of  system  maturity.  In  effect,  this  research  would 
hope  to  provide  a  contextual  decision  making  framework  for  effectively  using  the  TRL 
and  IRL  metrics  to  reduce  the  risk  associated  with  investing  in  immature  technologies 
(GAO  2005).  The  significance  of  this  research  lies  in  presenting  a  framework  for 
determining  component  maturity,  which  can  be  used  by  decision  makers  to  evaluate  the 
maturity  of  a  system.  The  information  presented  in  the  framework  is  not  intended  to  be 
used  as  a  “check  the  box”  event,  instead,  it  is  supposed  to  serve  as  a  platform  to  select 
models  that  can  be  used  to  harvest  information  for  making  more  informed  decisions  on 
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technology  and  integration  maturity.  It  is  expected  that  the  development  of  an 
assessment  platform  based  on  a  set  of  rules,  guidelines  and  ontology  for  consistency, 
repeatability,  and  traceability  will  allow  for  a  more  objective  approach  to  maturity 
assessment.  We  reiterate,  this  research  seeks  to: 

[1]  Identity  the  systems  engineering  architectural  artifacts  that  support  the 
assessment  of  a  technology  maturity  (via  Technology  Readiness  Levels), 
integration  maturity  (via  Integration  Readiness  Levels),  and  likewise  system 
maturity  (via  System  Readiness  Levels); 

[2]  Correlate  systems  engineering  architectural  artifacts  to  supported  views  and 
artifacts  within  the  DoDAF  that  enable  TRL  and  IRL  assessment;  and 

[3]  Develop  a  maturity  assessment  tool  that  works  with  standard  industry  SE 
architecture  tools  (e.g.  Sparx  Enterprise  Architect). 


3  Background 


3.1  Metrics 

Metrics  act  are  indicators  to  measure  the  attribute  of  an  object  of  interest  in  order  to 
make  more  informed  decisions  (Jacoby  and  Luqi  2007).  The  use  of  metrics  in  the  realms 
of  project  management  and  system  development  and  operational  sustainment  are  a 
proven  and  successful  practice  (Gove,  Sauser  et  al.  2007).  Dowling  and  Pardoe  (2005) 
lists  four  rules  required  to  create  a  successful  metric: 

1)  The  way  the  value  is  used  should  be  clear; 

2)  The  data  to  be  collected  for  the  metric  sould  be  easily  understood  and  easy 
to  collect; 

3)  The  way  of  deriving  the  value  from  the  data  should  be  clean  and  as  simple 
as  possible;  and 

4)  Those  for  whom  the  use  of  the  metric  implies  additional  cost  should  see  as 
much  direct  benefit  as  possible. 

Based  on  these  rules  we  can  then  define  metrics  into  two  classifications:  Descriptive  or 
Prescriptive  (Fan  and  Yih  1994;  Tervonen  and  Iisakka  1996;  Harjumaa,  Tervonen  et  al. 
2008).  Descriptive  metrics,  or  sometimes  referred  to  as  hard  metrics,  can  be  objectively 
measured,  are  quantifiable,  and  have  minimal  variability  when  used  between  observers. 
For  example,  the  height  of  an  individual,  proportion  of  telephone  calls  answered,  or 
machine  downtime.  On  the  other  hand,  prescriptive  metrics,  or  soft  measures,  are  those 
which  are  qualitative,  judgmental,  subjective,  and  based  on  perceptual  data.  For 
example,  customers'  satisfaction  with  speed  of  service  or  managers'  assessment  of  staff 
attitude  towards  customers  (Dowling  and  Pardoe  2005).  With  prescriptive  metric, 
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when  not  used  in  the  proper  context,  multiple  observers  can  assess  the  same  problem 
and  yield  significantly  different  results. 

Within  systems  engineering  we  have  come  to  rely  on  prescriptive  metrics  for  making 
managerial  and  at  times  engineering  decisions  because  there  is  limited  descriptive  data. 
Prescriptive  metrics  are  strongly  based  on  human  interpretation  that  can  be  influenced 
by  personal  biases  and  preferences.  Lee  and  Shin  (Lee  and  Shin  2000)  found  that 
egocentric  biases  and  personal  goals  play  a  large  role  in  human  beings’  evaluation 
process.  Since  such  cognitive  bias  is  involved  in  assessment,  subjectivity  is  more  or  less 
inherent  in  our  estimation  and  it  is  very  hard  to  avoid  its  influence  (Yan,  Xu  et  al.  2006). 

Prescriptive  metrics  are  vital  in  providing  fuller  insight  that  some  descriptive  metrics 
cannot,  yet  perspective  metrics  have  been  wrongfully  considered  as  less  important. 
While  descriptive  metrics  take  into  account  more  qualitative  factors,  it  is  possible  to 
bridge  and  attempt  to  quantity  qualities  that  are  difficult  to  assess  with  both  collectively. 
This  research  makes  strides  to  moving  the  prescriptive  metrics  of  TRL  and  IRL  closer  to 
a  descriptive  state.  But,  with  any  prescriptive  metric,  for  effective  use,  there  is  a  need  to 
understand  their  boundaries  and  limitations. 


3.2  Readiness  Levels 

The  success  behind  using  TRL  has  opened  up  the  path  for  researchers  to  identify 
alternative  readiness  (maturity)  levels  that  will  complement  TRL.  TRL  has  been 
implemented  and  modified  since  the  early  1990^  in  government  programs  and  has 
proved  to  be  a  beneficial  metric  in  assessing  the  risks  associated  with  a  developing  or 
acquired  technology  (see  Table  1  for  the  definitions  and  description  of  TRL  levels).  Just 
as  the  ways  that  agencies  or  organizations  have  adopted  the  TRL  metric  or  created  new 
readiness  levels  have  been  diverse,  so  have  the  ways  that  they  employ  these  metrics 
(Tan,  Sauser  et  al.  2011).  Graettinger,  et  al.  (Graettinger,  Garcia  et  al.  2002)  reports  that 
approaches  for  readiness  level  implementation  among  agencies  are  quite  broad,  which 
range  from  a  formal  software  tool  to  more  informal  face-to-face  discussions  between 
stakeholders. 
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Table  1  Technology  Readiness  Level 


TRL 

Definition 

Description 

1 

Basic  principles  observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be 
translated  into  applied  research  and  development  (R&D).  Examples  might 
include  paper  studies  of  a  technology’s  basic  properties. 

2 

Technology  concept  and/or 
application  formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical  applications 
can  be  invented.  Applications  are  speculative,  and  there  may  be  no  proof  or 
detailed  analysis  to  support  the  assumptions.  Examples  are  limited  to 
analytic  studies. 

3 

Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept. 

Active  R&D  is  initiated.  This  includes  analytical  studies  and  laboratory 
studies  to  physically  validate  the  analytical  predictions  of  separate  elements 
of  the  technology.  Examples  include  components  that  are  not  yet  integrated 
or  representative. 

4 

Component  and/or  breadboard 
validation  in  a  laboratory  environment. 

Basic  technological  components  are  integrated  to  establish  that  they  will 
work  together.  This  is  relatively  “low  fidelity”  compared  with  the  eventual 
system.  Examples  include  integration  of  “ad  hoc”  hardware  in  the 
laboratory. 

5 

Component  and/or  breadboard 
validation  in  a  laboratory  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic 
technological  components  are  integrated  with  reasonably  realistic 
supporting  elements  so  they  can  be  tested  in  a  simulated  environment. 
Examples  include  “high-fidelity”  laboratory  integration  of  components. 

6 

System/subsystem  model  or  prototype 
demonstration  in  a  relevant 
environment. 

Representative  model  or  prototype  system,  which  is  well  beyond  that  of 

TRL  5,  is  tested  in  a  relevant  environ-  ment.  Represents  a  major  step  up  in 
a  technology’s  demonstrated  readiness.  Examples  include  testing  a 
prototype  in  a  high-fidelity  laboratory  environment  or  in  a  simulated 
operational  environment. 

7 

System  prototype  demonstration  in  an 
operational  environment. 

Prototype  near  or  at  planned  operational  system.  Represents  a  major  step 
up  from  TRL  6  by  requiring  demonstration  of  an  actual  system  prototype  in 
an  operational  environment  (e.g.,  in  an  air-  craft,  in  a  vehicle,  or  in  space). 

8 

Actual  system  completed  and  qualified 
through  test  and  demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under  expected 
conditions.  In  almost  all  cases,  this  TRL  represents  the  end  of  true  system 
development.  Examples  include  developmental  test  and  evaluation  (DT&E) 
of  the  system  in  its  intended  weapon  system  to  deter-  mine  if  it  meets 
design  specifications. 

9 

Actual  system  proven  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under  mission 
conditions,  such  as  those  encountered  in  operational  test  and  evaluation 
(OT&E).  Examples  include  using  the  system  under  operational  mission 
conditions. 

After  the  DoD  began  its  adoption  of  the  TRL  metric,  much  effort  was  invested  in 
applying  the  metric  to  technologies  in  ongoing  programs  and  projects.  To  support  this,  a 
Technology  Readiness  Assessment  (TRA)  Deskbook  has  provided  the  guidance  for 
performing  technology  maturity  assessments  prior  to  incorporating  these  technologies 
into  systems  in  defense  programs.  In  addition,  TRL  calculators  have  been  created 
(Nolte,  Kennedy  et  al.  2003)  as  a  tool  in  technology  maturity  assessment.  For  each  of 
these  efforts,  guidance  such  as  readiness  level  descriptions  and/or  checklists  have  been 
used  individually  or  in  combination. 

However,  critics  of  the  TRL  system  have  argued  that  the  TRL  metric  combines  many 
dimensions  of  technology  readiness  into  one  metric  (Smith  2004).  Kaplan  and  Norton 
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have  said,"  what  you  measure  is  what  you  get  (Kaplan  and  Norton  2010),  hence  the 
failure  to  consider  an  attribute  can  lead  to  inaccurate  assessment. 

Given  the  emerging  needs  for  a  measure  of  system  readiness,  in  2006  the  SysDML  at 
Stevens  Institute  of  Technology  presented  the  concept  of  a  System  Readiness  Level  for 
managing  system  development  (Sauser,  Verma  et  al.  2006).  As  a  result,  in  2007  the 
SysDML  in  collaboration  with  the  US  Navy  PMS  420/SPA  WAR  and  Northrop  Grumman 
Corporation  were  chartered  to  define  a  system  maturity  scale  and  supporting 
methodology.  The  core  requirements  included  that  the  scale  must  be  robust,  repeatable, 
and  agile  so  outputs  could  not  only  be  trusted  and  replicated,  but  that  the  methodology 
as  a  whole  be  easily  transferred  to  a  variety  of  different  applications  and  architectures. 

In  response  to  this  challenge,  the  concept  of  a  System  Readiness  Level  (SRL)  that  would 
incorporate  a  TRL  and  an  Integration  Readiness  Level  (IRL)  was  developed  as  depicted 
in  Figure  1  (Sauser,  Verma  et  al.  2006) 


The  System 


Technology 
Level  1  TRL) 


*  7) 


IntegiaUon 
lUadiMti 
Level  (IRL) 


♦ 

SRL  = /(TRL,  IRL) 

Figure  1  System  Readiness  Level 


Similar  to  TRL,  the  IRL  is  defined  as  a  series  of  levels  that  articulate  the  key  maturation 
milestones  for  integration  activities  (see  Table  2  for  the  definitions  and  description  of 
IRL  levels).  The  introduction  of  an  IRL  to  the  assessment  process  not  only  provides  a 
check  as  to  where  a  technology  is  on  an  integration  readiness  scale  but  also  presents  a 
direction  for  improving  integration  with  other  technologies.  Just  as  a  TRL  is  used  to 
assess  the  risk  associated  with  developing  technologies,  the  IRL  is  designed  to  assess  the 
risk  associated  with  integrating  these  technologies.  For  more  details  on  the  formulation 
of  the  IRL  see  (Sauser,  Gove  et  al.  2010). 
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Table  2:  Integration  Readiness  Level 


IRL 

Definition 

Description 

9 

Integration  is  Mission  Proven  through 
successful  mission  operations. 

IRL  9  represents  the  integrated  technologies  being  used  in  the  system 
environment  successfully.  In  order  for  a  technology  to  move  to  TRL  9 
it  must  first  be  integrated  into  the  system,  and  then  proven  in  the 
relevant  environment,  so  attempting  to  move  to  IRL  9  also  implies 
maturing  the  component  technology  to  TRL  9. 

8 

Actual  integration  completed  and  Mission 
Qualified  through  test  and  demonstration, 
in  the  system  environment. 

IRL  8  represents  not  only  the  integration  meeting  requirements,  but 
also  a  system-level  demonstration  in  the  relevant  environment.  This 
will  reveal  any  unknown  bugs/defect  that  could  not  be  discovered  until 
the  interaction  of  the  two  integrating  technologies  was  observed  in  the 
system  environment. 

7 

The  integration  of  technologies  has  been 
Verified  and  Validated  and  an 
acquisition/insertion  decision  can  be 
made. 

IRL  7  represents  a  significant  step  beyond  IRL  6;  the  integration  has  to 
work  from  a  technical  perspective,  but  also  from  a  requirements 
perspective.  IRL  7  represents  the  integration  meeting  requirements 
such  as  performance,  throughput,  and  reliability. 

6 

The  integrating  technologies  can  Accept, 
Translate,  and  Structure  Information  for 
its  intended  application. 

IRL  6  is  the  highest  technical  level  to  be  achieved,  it  includes  the 
ability  to  not  only  control  integration,  but  specify  what  information  to 
exchange,  unit  labels  to  specify  what  the  information  is,  and  the  ability 
to  translate  from  a  foreign  data  structure  to  a  local  one. 

5 

There  is  sufficient  Control  between 
technologies  necessary  to  establish, 
manage,  and  terminate  the  integration. 

IRL  5  simply  denotes  the  ability  of  one  or  more  of  the  integrating 
technologies  to  control  the  integration  itself;  this  includes  establishing, 
maintaining,  and  terminating. 

4 

There  is  sufficient  detail  in  the  Quality  and 
Assurance  of  the  integration  between 
technologies. 

Many  technology  integration  failures  never  progress  past  IRL  3,  due  to 
the  assumption  that  if  two  technologies  can  exchange  information 
successfully,  then  they  are  fully  integrated.  IRL  4  goes  beyond  simple 
data  exchange  and  requires  that  the  data  sent  is  the  data  received 
and  there  exists  a  mechanism  for  checking  it. 

3 

There  is  Compatibility  (i.e.  common 
language)  between  technologies  to 
orderly  and  efficiently  integrate  and 
interact. 

IRL  3  represents  the  minimum  required  level  to  provide  successful 
integration.  This  means  that  the  two  technologies  are  able  to  not  only 
influence  each  other,  but  also  communicate  interpretable  data.  IRL  3 
represents  the  first  tangible  step  in  the  maturity  process. 

2 

There  is  some  level  of  specificity  to 
characterize  the  Interaction  (i.e.  ability  to 
influence)  between  technologies  through 
their  interface. 

Once  a  medium  has  been  defined,  a  “signaling”  method  must  be 
selected  such  that  two  integrating  technologies  are  able  to  influence 
each  other  over  that  medium.  Since  IRL  2  represents  the  ability  of  two 
technologies  to  influence  each  other  over  a  given  medium,  this 
represents  integration  proof-of-concept. 

1 

An  Interface  between  technologies  has 
been  identified  with  sufficient  detail  to 
allow  characterization  of  the  relationship. 

This  is  the  lowest  level  of  integration  readiness  and  describes  the 
selection  of  a  medium  for  integration. 

With  the  ability  to  assess  both  the  technologies  and  integration  elements  along  a 
numerical  maturation  scale,  the  next  challenge  was  to  develop  a  metric  that  could  assess 
the  maturity  of  the  entire  system  under  development.  Therefore,  the  SRL  was  developed 
that  could  incorporate  both  TRLs  and  IRLs  system  maturity  assessment  (Sauser, 
Ramirez-Marquez  et  al.  2008).  The  rationale  behind  the  SRL  is  that  in  the  development 
lifecycle,  one  would  be  interested  in  addressing  the  following  considerations: 

•  Quantifying  how  a  specific  technology  is  being  integrated  with  every  other 
technology  to  develop  the  system. 

•  Providing  a  system-wide  measurement  of  readiness. 
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For  a  detailed  description  of  the  SRL  methodology  see  Sauser,  B.,  J.E.  Ramirez- 
Marquez,  D.  Nowicki,  A.  Deshmukh,  and  M.  Sarfaraz.  Development  of  Systems 
Engineering  Maturity  Models  and  Management  Tools.  Systems  Engineering  Research 
Center  Final  Technical  Report  2011-TR-014,  January  2011  or  visit 
http://www.SvsDML.com 


3.3  System  Architecture  and  DoDAF 

The  International  Council  on  Systems  Engineering  (INCOSE)  defines  system 
architecture  as  “the  arrangement  of  elements  and  subsystems  and  the  allocation  of 
functions  to  them  to  meet  system  requirements”  (INCOSE  2007).  System  architectures 
can  help  systems  engineers  to  examine  a  system  from  various  perspectives,  and  for  that, 
architectures  help  decision  makers  to  reason  about  a  problem  (Dimov,  Stankov  et  al. 
2009).  In  support  of  this,  modeling  is  used  to  improve  communication  and  to  involve 
stakeholders,  developers,  integrators,  vendors,  and  testers  in  the  process  (Friendenthal 
2008). 

A  National  Research  Council  (2008)  study  recently  highlighted  that  architecture  can 
mitigate  internal  and  external  system  complexity  risk  by  partitioning  the  system  into 
separately  definable  procurable  parts  and  recommending  a  rigorous  development  of 
systems  architecture  early  on  in  the  program.  Much  of  this  information  is  efficiently 
and  effectively  conveyed  and  managed  via  architecture  products  (Hughes  2010).  In  the 
mid  1990s,  the  DoD  determined  that  a  common  approach  was  needed  for  describing  its 
architectures,  so  DoD  systems  could  efficiently  communicate  and  interoperate  during 
joint  and  multinational  operations  (Sibbald  2004).  This  need  led  to  the  introduction  of 
the  Command,  Control,  Communication,  Computers,  and  Intelligence,  Surveillance  and 
Reconnaissance  (C4ISR)  architectural  framework.  Subsequently,  further  revisions  of 
C4ISR  led  to  version  1.0  of  DoDAF  released  in  2003.  Ultimately,  DoD  directives  resulted 
in  the  official  use  of  DoDAF  1.0.  The  next  version,  DoDAF  1.5,  was  published  in  2007 
and  incorporated  net-centric  concepts  (DoDAF  2007).  Version  2.0  (DM2)  was  released 
in  2009,  placing  the  focus  on  architectural  data  rather  than  developing  products 
(DoDAF  2009).  DoDAF  continues  to  evolve,  but  for  the  purposes  of  this  research  we 
focused  on  DoDAF  2.0. 

There  are  a  number  of  notable  changes  from  previous  version  of  DoDAF  (1.0/1. 5)  to 
DM2.  For  example: 

•  DM2  does  not  require  all  DoDAF-described  models  to  be  created.  Key  process 
owners  have  the  responsibly  to  decide  which  activity  model  is  created,  but  once 
that  is  selected,  a  necessary  set  of  data  for  that  activity  model  is  required 
(Department  of  Defense  2009). 
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•  Another  feature  introduced  in  DM2  was  Fit-for- Purpose  (FFP)  models.  FFP 
models  are  useful  in  decision  making,  and  enable  the  architect  to  focus  on 
collecting  and  creating  views  that  are  necessary  for  the  decision  maker’s 
requirements,  and  focusing  the  architecture  to  align  to  the  decision-maker’s 
needs 


•  The  DoDAF  2.0  describes  the  technical  aspects  of  data  collection  and 
presentation,  organizer  thought  the  DM2,  enabling  the  requirements  of 
architecture  stakeholders  and  their  viewpoints  to  be  realized  through  both 
federation  efforts,  and  data  sharing.  The  DM2  defines  architectural  data  elements 
and  enables  the  integration  and  federation  of  Architectural  Descriptions”(DoD 
Architecture  Framework ).  The  DM2  provides  information  needed  to  collect, 
organize,  and  store  data  in  a  way  easily  understood.  The  presentation  description 
of  various  types  of  views  in  Volumes  1  and  2  provide  the  guidance  for  developing 
graphical  representations  of  that  data  that  is  useful  in  defining  acquisition 
requirements  under  the  DoD  Instruction  5000-series. 

Aside  from  DoDAF  2.0’s  new  features  that  can  help  in  acquisition  processes  and 
technology  management,  researchers  have  studied  using  DoDAF  in  technology 
management.  Dimov,  Stankov,  et  al.  (2009)  presented  an  architecture-oriented 
modeling  approach  to  assist  in  acquisition  systems  for  one  of  Bulgaria’s  force- 
management’s  subsystems.  Hughes  (2010)  from  the  Air  Force  Institute  of  Technology 
used  a  concept  maturity  model  to  help  to  uncover  the  unknowns  that  plague  a  system 
development.  Hughes  suggested  using  maturity  elements  to  assess  and  mature  a 
concept  at  a  given  decision  point.  The  limitation  in  this  research  is  that  an  explanation 
in  the  level  of  detail  is  required  for  each  maturity  element.  Scharch  and  Homan  (2011) 
latter  examined  the  applicability  and  validity  of  Hughes  framework  through  a  three 
tiered  methodology,  and  also  took  an  improvement  approach  to  the  framework.  Philips 
(2010)  introduced  Human  Readiness  Level  to  complement  TRL  in  program  risk 
management  structures,  and  synthesized  the  technical  details  of  the  Human  View  in 
relation  to  DoDAF.  Although  there  are  many  systems  architecture  platforms  that  can 
support  maturity  assessment,  this  research  utilizes  the  features  of  the  DoDAF  2.0 
models. 


4  Research  Results 


As  stated,  this  research  had  three  objectives: 

[1]  Identity  the  systems  engineering  architectural  artifacts  that  support  the 
assessment  of  a  technology  maturity  (via  Technology  Readiness  Levels), 
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integration  maturity  (via  Integration  Readiness  Levels),  and  likewise  system 
maturity  (via  System  Readiness  Levels); 

[2]  Correlate  systems  engineering  architectural  artifacts  to  supported  views  and 
artifacts  within  the  DoDAF  that  enable  TRL  and  IRL  assessment;  and 

[3]  Develop  a  maturity  assessment  tool  that  works  with  standard  industry  SE 
architecture  tools  (e.g.  Sparx  Enterprise  Architect). 

Section  4.1  will  describe  the  results  from  [1]  and  [2]  and  Section  4.2  will  describe  the 
results  of  [3].  The  actual  products  of  [1]  and  [2]  can  be  found  in  Appendix  A  and  for  [3] 
the  products  can  be  acquired  by  contacting  Brian  Sauser  at  bsauser(5> stevens.edu  or 
visiting  http://www.SvsDML.com 


4.1  Mapping  Readiness  Levels  to  DoDAF 

In  the  process  of  mapping  maturity  elements  of  a  given  readiness  level  to  architecture 
artifacts,  it  is  imperative  that  the  selection  of  models  can  address  a  particular  question 
or  questions.  An  obstacle  in  this  process  is  to  choose  models  from  the  large  number  of 
standard  models  in  DM2.  To  address  this  we  followed  a  process  that  would  allow  us  to 
reduce  the  number  of  choices  down  to  a  smaller  subset.  That  is,  a  subset  where  its 
elements  are  more  likely  to  contain  the  models  that  can  address  a  particular  question. 
The  approach  this  research  used  to  pair  TRL/IRL  maturity  criteria  to  DoDAF  artifacts 
was  achieved  in  four  steps.  This  is  shown  in  Figure  2  and  described  below. 

The  first  step  was  the  extraction  of  all  TRL  and  IRL  decision  criteria.  We  used  the  TRL 
Calculator  and  the  IRL  decision  criteria  as  described  by  Sauser,  et  al.  (2010)  to  define 
these  criteria.  An  excel  spreadsheet  was  used  to  populate  all  the  decision  criteria 
questions.  When  possible,  composite  questions  with  multiple  parts  were  broken  down  to 
sub-questions  to  provide  a  more  direct  response.  While  the  TRL  Calculator  Tool 
provides  information  on  both  hardware  and  software  criteria  for  determining  a  TRL,  we 
focused  on  the  hardware  questions  only.  While  we  believe  hardware  and  software  on 
most  systems  is  inseparable,  we  wanted  to  eliminate  any  criteria  that  were  ambiguous. 
The  IRL  criteria  as  described  by  Sauser,  et  al.  (2010)  do  not  distinguish  between 
hardware  and  software. 
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Review  and  verify  selection 
and  mapping 


Compile  Readiness 
Level  Decision  Criteria 


Map  the  type  of  DM2  Conceptual 
Data  Models  to  Readiness  Level 
Decision  Criteria 


Select  Models  from  a  Subset 
list  of  all  DoDAF  models 


Figure  2:  Artifact  to  Readiness  Level  Mapping  Process 


The  second  step  was  to  determine  the  type  of  DM2  Conceptual  Data  Models  (CDM)  that 
can  address  each  readiness  level  decision  criteria.  The  CDM  defines  concepts  involving 
high-level  data  constructs  from  which  Architectural  Descriptions  are  created,  enabling 
executives  and  managers  at  all  levels  to  understand  the  data  basis  of  an  Architectural 
Description  (DoDAF  2009).  The  primitive  underlying  concept  of  using  DM2  is  to  group 
semantically  related  concepts  into  clusters.  Figure  3  below  shows  key  concepts  as  they 
are  grouped  into  three  categories  (Ways,  Means,  and  Ends)  to  facilitate  the 
identification  and  selection  of  architecture  models.  This  helped  to  build  Figure  3,  where 
a  matrix  determines  which  views  are  addressed  by  any  given  cluster.  There  are  some 
views  that  are  more  important  than  others,  but  there  are  times  that  to  answer  a 
question,  the  answers  are  distributed  amongst  different  views. 


Contract  Number:  H98230-08-D-01 71 

Report  No.  SERC-2012-TR-027 
March  2,  2012 


DO  004  TO  001  RT  027 


UNCLASSIFIED 

18 


UNCLASSIFIED 


Description  of  each  key  concept  as  it  relates  to  Maturity  Assessment 

Behavior 

Activity 

Work  that  tree* form*  input*  (Resource*)  into  output*  (Rasourofts)  or  changu*  ttuur  statu 

Meavure 

The  magnitude  of  noaic  aft*  ibule  nt  an  indlviduil 

Rule 

Agree  merit 

A  consent  among  parties  regarding  the  term*  and  onnditkm*  of  activities 

Standard 

A  formal  agreement  documenting  accepted  specifications  or  criteria  for  product*,  processes,  procedures 

Links 

Relationship 

A  connection,  association,  or  involvement. 

Interaction 

The  interaction  between  two  component*  or  system*,  sending  data 

Resources 

System 

A  functionally,  physically,  end/or  behavior  ally  related  group  of  atiler  acting  nr  inter  dependent  atemanla. 

Service 

A  mechanism  to  enable  access  to  capabilities,  access  provided  using  interlace.  Uses  Resources 

Or  gAn17.1t  ion 

A  specific  real  world  assemblage  of  people  and  other  resoiaces  organized  for  an  on  going  purpose. 

Penon/Skill 

I  he  ability,  coming  from  one's  knowledge,  practice,  aptitude.  «tc_  to  do  something  well. 

Matetial/HW/SW 

Equipment,  apparatus  or  supplies  that  are  of  interest,  without  distinction  a*  to  application 

I 

Data 

Information  suitable  tor  communication,  interpretation,  or  processing  by  humans  or  machines 

!nfo/Dat» 

Information 

T  he  state  of  a  something  of  Interest  that  is  materialized  -  in  any  medium  or  form  -  and  onmmunicatod  or  received. 

Results 

Project /Tati 

Planned  act wdws  that  aim  to  Change  the  rtirt*  of  some  situation.  A  temporal  y  endeavor  to  create  Resources  or  Desired  Cftects 

Capability 

the  ability  to  achieve  a  Desired  L  fleet  under  specified  (performance)  standards  and  conditions,  by  usmg  ways  and  means 

Goal/FMrct 

How  goafs,  visions,  objectives,  and  effects  relate  and  bear  on  architectures.  A  desired  state  of  a  Resource 

Means 


Figure  3  Most  popular  DM2  Conceptual  Data  Model  concepts  used  to  facilitate  the  collection  and 
usage  of  architecture  related  data 


The  goal  of  the  third  and  fourth  steps  were  to  select  models  from  a  subset  list  of  all 
DoDAF  models.  This  was  achieved  by  a  correlation  table  matrix  of  the  DM2  clusters  to 
the  list  of  DoDAF  models  (see  example  in  Figure  4).  This  step  helped  to  obtain  a  subset 
list  of  DoDAF  described  models,  which  may  or  may  not  address  the  question.  A  decision 
maker  with  the  knowledge  of  DoDAF  models  can  select  the  appropriate  models  from  the 
subset  list.  Regardless,  a  smaller  list  improves  the  chances  of  the  identification  of 
models  that  may  have  otherwise  been  overlooked.  The  resulting  mapping  of  TRL  and 
IRL  decision  criteria  to  DoDAF  models  can  be  found  in  Appendix  A. 
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Figure  4  DM2  clusters  to  the  list  of  DoDAF  models2 


2  J.  Martin,  Architecture  Frameworks  &  Modeling  Part  2:  Architecture  Development,  Liberty  Chapter, 
Mar  31  &  Apr  1,  2011 
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4.2  SRL  Tools  Development 

Many  of  the  traditional  tools  and  services  are  inadequate  to  deal  with  the  increasing 
complexity  of  systems.  As  a  result,  systems  engineers  have  opted  for  using  more 
dedicated  tools  (e.g.  Rational  Rhapsody,  Sparx  Enterprise  Architect)  to  consider  all 
different  types  of  information  relevant  to  the  decision  making  process.  Thus,  as  our 
understanding  of  a  technology  changes,  so  should  the  tools  we  use  to  analyze  the  system 
these  technologies  and  supporting  integrations  comprise.  The  goal  of  this  phase  of  the 
research  was  to  develop  a  calculator  that  would  allow  for  the  computation  of  a  SRL 
through  a  standard  industry  systems  architecture  tool  (e.g.  Sparx  Enterprise  Architect). 
The  result  was  a  plug-in  SRL  calculator  that  would  work  in  conjunction  with  Sparx 
Enterprise  Architect  (EA).  In  addition,  because  the  calculator  uses  an  export  xmi  file,  it 
also  has  the  capability  with  limited  modifications  to  be  used  with  other  systems 
architecting  tools  (e.g.  IBM  Rhapsody). 

The  SRL  calculator  adheres  to  the  SRL  methodology  and  thus  the  SRL  is  a  product  of 
the  TRL  and  IRL  values.  The  SRL  calculator  extracts  TRL  and  IRL  information  from  a 
systems  architecture  model  and  reports  the  SRL  value,  the  supporting  Integrated 
Technology  Readiness  Level  (ITRL)  values,  as  well  as  a  graphical  aid  to  present  the 
calculated  values  in  relation  to  standard  systems  engineering  lifecycle  phases.  To 
acquire  a  copy  of  the  tool  and  supporting  documentation,  contact  Brian  Sauser  at 
bsauser(5) stevens.edu  or  see  http://www.SvsDML.com. 

Also  as  part  of  this  a  second  SRL  calculator  was  created  that  is  not  dependent  on  a 
systems  architecting  tool  for  TRL  and  IRL  data  inputs.  This  SRL  calculator  is  HTML 
based  and  can  be  run  from  any  web  browser.  A  copy  of  the  calculator  can  be  found  at 
http://www.SvsDML.com 


5  Conclusions 


Architecture  development  efforts  need  to  be  in  line  with  the  goals  and  objectives  of  the 
project,  hence  the  decision  to  invest  in  the  development  of  a  particular  architecture 
model  can  depend  on  a  variety  of  factors.  In  integrated  architectures,  the  choice  to 
invest  in  a  designated  model  may  depend  on  the  rigor  of  the  detail  placed  in  alternative 
models  which  contain  similar  information.  Hence,  the  selection  of  models  differs  from 
project  to  project,  and  is  subject  to  the  decision  maker’s  discretion. 

As  mentioned  earlier,  the  results  of  this  study  lists  the  DoDAF  described  models  that 
can  support  maturity  assessment.  The  results  of  this  study  can  introduce  a  new  facet 
where  maturity  type  information  can  be  introduced  to  the  system  architecture. 
However,  there  is  no  need  to  make  these  views  universal  to  all  programs.  It  is  not  an 
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ideal  practice  to  interrupt  the  natural  engineering  and  architecting  process  by  focusing 
on  specific  views,  regardless  of  their  technical  necessity  for  solving  the  problems  at 
hand. 

The  identification  of  more  models  and  maturity  artifacts  is  advantageous  to  this 
research,  as  it  will  provide  for  more  architectural  maturity  artifacts,  improving 
technology  and  integration  maturity  assessment.  However,  one  should  be  careful  to 
notice  that  supplying  DoDAF  views  of  an  architecture  can  give  a  false  impression  of 
“architecting”  being  complete  (Bergey,  Blanchette  et  al.  2009). 

Throughout  the  development  of  a  product,  many  of  the  ideas  will  need  to  be  updated  in 
later  stages  of  the  program.  Hence,  as  more  information  about  the  project  becomes 
available,  this  information  can  be  updated  in  the  system  architecture  model.  In  other 
words,  system  architecting  is  an  iterative  process. 

Generally  what  we  would  expect  to  see  in  a  system  architecture  would  depend  on  what 
we  would  like  to  extract  from  it.  To  the  interest  of  this  research  is  information  on  the 
lifecycle,  together  with  the  decomposition  into  subsystems  and  their  interrelations.  For 
the  purpose  of  this  research,  what  is  most  important  is  having  a  repository  of  data  and 
models,  and  being  able  to  use  them  for  analysis  and  decision  making. 
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Appendix  A:  Readiness  Levels  to  DoDAF 


A.l  TRL  TO  DoDAF 


TRL  Criteria 

Basic  principles  observed  and  reported 


Do  rough  calculations  support  the  concept? 

Do  basic  principles  (physical,  chemical,  mathematical)  support  the  concept? 
Does  it  appear  the  concept  can  be  supported  by  hardware? 

Are  the  hardware  requirements  known  in  general  terms? 

Do  paper  studies  confirm  basic  scientific  principles  of  new  technology? 

Have  mathematical  formulations  of  concepts  been  developed? 

Have  the  basic  principles  of  a  possible  algorithm  been  formulated? 

Have  scientific  observations  been  reported  in  peer  reviewed  reports? 

Has  a  sponsor  or  funding  source  been  identified? 

Has  a  scientific  methodology  or  approach  been  developed? 

TRL  Criteria 

2  Technology  concept  and/or  application  formulated 


Has  potential  system  or  component  applications  been  identified? 

Have  paper  studies  confirmed  system  or  component  application  feasibility? 

Is  the  end  user  of  the  technology  known? 

Has  an  apparent  design  solution  been  identified? 

Have  the  basic  components  of  the  technology  been  identified? 

Has  the  user  interface  been  defined? 

Have  technology  or  system  components  been  at  least  partially  characterized? 

Have  performance  predictions  been  documented  for  each  component? 

Has  a  customer  expressed  interest  in  application  of  technology? 

Has  a  functional  requirements  generation  process  been  initiated? 

Does  preliminary  analysis  confirm  basic  scientific  principles? 

Have  draft  functional  requirements  been  documented? 

Have  experiments  validating  the  concept  been  performed  with  synthetic  data? 

Has  a  requirements  tracking  system  been  initiated? 

Are  basic  scientific  principles  confirmed  with  analytical  studies? 

Have  results  of  analytical  studies  been  reported  to  scientific  journals,  etc.? 

Do  all  individual  parts  of  the  technology  work  separately?  (No  real  attempt  at  integration) 
Is  the  hardware  that  the  software  will  be  hosted  on  available? 

Are  output  devices  available? 


DoDAF-Described  Models 


CV-1,3 

OV-6b 

CV-1 

SV-3 

OV-2 

CV-6 

OV-1,2 

AV-1 

FFP 

AV-1 

AV-1 

FFP 

AV-1 

AV-1 

CV-5 

AV-1 

DoDAF-Described  Models 


OV-1 

PV-2 

SV-1 

AV-1 

OV-1 

OV-2, 5a, 5b 

SV-1 ,2,6 

OV-1, 2, 4 

SV-1 ,2,4, 

StdV-1 ,2 

SV-1 ,2,4 

SV-3, 4, 5a, 6 

SV-9 

CV-6 

OV-5a,5b 

SV-3, 4, 5a, 6 

PV-2 

AV-1 

SV-4 

CV-1 

SV-4 

CV-1 

PV-2 

StdV-1 

AV-1 

StdV-1 ,2 

AV-1 

AV-1 

StdV-1 

OV-2, 5a, 5b 

OV-2, 5a, 5b 
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TRL  Criteria 

Analytical  and  experimental  critical  function  and/or  characteristic  proof-of-concept 


Have  predictions  of  components  of  technology  capability  been  validated? 

Have  analytical  studies  verified  performance  predictions  and  produced  algorithms? 

Can  all  science  applicable  to  the  technology  be  modeled  or  simulated? 

Have  system  performance  characteristics  and  Measures  been  documented? 

Do  experiments/M&S  validate  performance  predictions  of  technology  capability? 

Does  basic  laboratory  research  equipment  verify  physical  principles? 

Do  experiments  verify  feasibility  of  application  of  technology? 

Do  experiments/M&S  validate  performance  predictions  of  components  of  technology  capability? 

Has  customer  representative  to  work  with  R&D  team  been  identified? 

Is  customer  participating  in  requirements  generation? 

Have  cross-technology  effects  (if  any)  been  identified? 

Have  design  techniques  been  identified  and/or  developed? 

Do  paper  studies  indicate  that  technology  or  system  components  can  be  integrated? 

Has  Technology  Transition  Agreement  (TTA)  including  possible  TRL  for  transition  been  drafted? 

Are  the  technology/system  performance  metrics  established? 

Have  scaling  studies  been  started? 

Have  technology/system  performance  characteristics  been  confirmed  with  representative  data  sets? 
Do  algorithms  run  successfully  in  a  laboratory  environment,  possibly  on  a  surrogate  processor? 
Have  current  manufacturability  concepts  been  assessed? 

Can  key  components  needed  for  breadboard  be  produced? 

Has  analysis  of  alternatives  been  completed? 

Has  scientific  feasibility  of  proposed  technology  been  fully  demonstrated? 

Does  analysis  of  present  technologies  show  that  proposed  technology/system  fills  a  capability  gap? 

TLR  Criteria 

Component  and/or  breadboard  validation  in  laboratory  environment 


Low  fidelity  hardware  technology  system  integration  and  engineering  completed  in  a  lab  environment 
Technology  demonstrates  basic  functionality  in  simplified  environment 
Scaling  studies  have  continued  to  next  higher  assembly  from  previous  assessment 
BMDS  mission  enhancement(s)  clearly  defined  within  goals  of  study. 

Integration  studies  have  been  started. 

Draft  conceptual  hardware  and  software  designs. (provide  copy  of  documentation_ 

Some  software  components  are  available. 

Piece  parts  and  components  in  pre-production  form  exist.  Provide  documentation. 

Production  and  integration  planning  have  begun.  Documentation 
Performance  metrics  have  been  established 
Cross  technology  issues  have  been  fully  identified. 

Design  techniques  have  been  defined  to  the  point  where  : 

Begin  discussions/negotiations  of  Technology  Transition  Agreement 


DoDAF-Described  Models 


CV-2 

PV-1 ,2 

CV-2 

CV-2 

SV-4,7 

SvcV-4,7 

SvcV-7 

CV-2 

CV-2 

CV-2 

AV-1 

CV-6 

PV-1 

SV-3 

SV-2 

OV-5a,5b 

SV-5a,5b 

FFP 

CV-2 

CV-3 

SV-9 

SV-7 

SvcV-7 

PV-2 

OV-1 

PV-1 

SV-1 

PV-2 

SV-7 

CV-5 

PV-2, 3 

StdV-1 

SV-1 

PV-2, 3 

OV-6a 

PV-2 

SV-4 

CV-1,2 

SV-8 

DoDAF-Described  Models 

PV-2 

SV-1  ,2, 3, 4, 6 

FFP 

PV-1 ,2 

FFP 

PV-1 ,2,3 

CV-2, 4, 6 

PV-1 

FFP 

SV-1 ,2, 3, 4, 6 

OV-4,5a,5b,6a,6b 

PV1 ,2 

CV-3 

OV  2,3 

PV-1 

SV-3 

SV-1 ,2,3,4,6,10a 

CV-3 

SV-7 

SvcV-7 

SV-1 ,2,3,4,6,10a 

OV-3,4, 

PV-2 

CV-3 

SV-9 
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TRL 

5 


TRL 

6 


Criteria  DoDAF-Described  Models 

Component  and/or  breadboard  validation  in  relevant  environment 


High  fidelity  lab  integration  of  hardware  system  completed  and  ready  for  testing  in  realistic  simulated  environment. 
Preliminary  hardware  technology  engineering  report  completed 

Detailed  design  drawings  have  been  completed.  Three  view  drawings  and  wiring  diagrams  have  been  submitted. 
Pre-production  of  hardware  available. 

Form,  fit,  function  for  application  has  begun  to  be  addressed  in  conjunction  with  end  user  and  development  of  staff 
Cross  technology  effects(if  any)  identified  and  established  through  analysis. 

Design  techniques  have  been  defined  to  the  point  where  largest  problems  defined. 

Scaling  studies  have  continued  to  next  higher  assembly  from  prev  assessment 
TTA  has  been  updated  to  reflect  data  in  items  1  thru  3,  5,  8. 

Criteria 

System/subsystem  model  or  prototype  demonstration  in  a  relevant  environment 


SV-1 ,2,3,4,6,10a 

CV-3  PV-2 

CV-3  PV-2 

CV-4,5  PV-1,2 

CV-3  PV-2 

SV-1 ,2,3,4,6,10a 

CV-4  PV-1 ,2 

SV-1 ,2,4,6,9,10a  PV-2 

CV-4,5  PV-1 ,2 


DoDAF-Described  Models 


Materials,  process,  design,  and  integration  methods  have  been  employed. 

Scaling  issues  that  remain  are  identified  and  supporting  analysis  is  complete.  SV-1 

Production  demonstrations  are  complete.  Production  issues  have  been  identified  and  major  ones  have  been  resolved. 

Some  associated  “Beta”  version  software  is  available. 

Most  pre-production  hardware  is  available. 

Draft  production  planning  has  been  reviewed  by  end  user  and  developer. 

Draft  design  drawings  are  nearly  complete. 

Integration  demonstrations  have  been  completed,  including  cross  technology  issue  Measurement  and  performance  characteristic  validations.  SV-1 
Have  begun  to  establish  an  interface  control  process. 

Collection  of  actual  maintainability,  reliability,  and  supportability  data  has  been  started. 

Representative  model  or  prototype  is  successfully  tested  in  a  high-  fidelity  laboratory  or  simulated  operational  environment. 

Hardware  technology  "system"  specification  complete. 

Technology  Transition  Agreement  has  been  updated  to  reflect  data  in  items  1  through  4,  7  through  9,  11  and  1 2. 


,2,3,4,6,9,10a 

PV-1 ,2,3 

SV-9 

CV-3 

OV  2,3 

CV-3 

SV-6 

CV-5,6 

PV-2 

,2,3,4,6,9,10a 

SV-2,3,8 

SvcV-2,3,8 

CV-1,5 

PV-2 

SV-4,5a 

SvcV-4,  5a 

CV-4,5 

PV-1,2 

FFP 


SV-2,3,5a 
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TRL 

Criteria 

DoDAF-Described  Models 

7 

System  prototype  demonstration  in  a  space  environment 

Material,  processes,  methods,  and  design  techniques  have  been  identified 

SV-4 

StdV-1 

Scaling  is  complete. 

PV-2 

Production  planning  is  complete. 

CV-5,7 

Pre-production  hardware  and  software  is  available  in  limited  quantities. 

PV-2 

SV-3 

Draft  design  drawings  are  complete. 

FFP 

PV-2 

Maintainability,  reliability,  and  supportability  data  growth  is  above  60%  of  total  needed  data. 

PV-2 

SV-7 

SvcV-7 

Hardware  technology  "system"  prototype  successfully  tested  in  a  field  environment. 

PV-2 

CV-3 

TRL 

Criteria 

DoDAF-Described  Models 

8 

Actual  system  completed  and  qualifie  through  test  and  demonstration 

Interface  control  process  has  been  completed 

Maintainability,  reliability,  and  supportability  data  collected,  and  has  been  completed. 

CV-1,5 

OV-4 

PV-2 

Hardware  technology  successfully  completes  developmental  test  and  evaluation. 

CV-1,5 

OV-2,3,4 

PV-2 

Hardware  technology  has  been  proven  to  work  in  its  final  form  and  under  expected  conditions. 

FFP 

OV-AII 

SV-ALL,  SvcV-AII 

TRL 

Criteria 

DoDAF-Described  Models 

9 

Actual  system  "flight  proven"  through  successful  mission  operations 

Hardware  technology  successfully  completes  operational  test  and  evaluation 

CV-ALL 

FFP 

OV-AII 

Training  plan  has  been  implemented. 

CV-1,5 

OV-4 

PV-2 

Supportability  plan  has  been  implemented. 

SV-1 ,2,3,4,6,10a 

Program  Protection  plan  has  been  implemented. 

AV-2 

CV-1,2,4 

PV-2 

Safety/Adverse  effects  issues  have  been  identified  and  mitigated. 

SV-1 ,2,3,4,6,10a 

Operation  concept  has  been  implemented  successfully. 

OV-  ALL 

PV-AII 

SV-AII 
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A.2  IRLtoDoDAF 


IRL  Criteria 

.  An  Interface  between  technologies  has  been  identified  with  sufficient  detail  to  allow  characterization  of  the  relationship. 


Principal  integration  technologies  have  been  identified 
Top-level  functional  architecture  and  interface  points  have  been  defined 
Availability  of  principal  integration  technologies  is  known  and  documented 
Integration  concept/plan  has  been  defined/drafted 
Integration  test  concept/plan  has  been  defined/drafted 

High-level  Concept  of  Operations  and  principal  use  cases  havebeen  defined/drafted 
Integration  sequence  approach/schedule  has  been  defined/drafted 
Interface  control  plan  has  been  defined/drafted 

Principal  integration  and  test  resource  requirements  (facilities, hardware,  software,  surrogates,  etc.)  have  been  defined/identified 
Integration  &  Test  Team  roles  and  responsibilities  have  been  defined 

IRL  Criteria 

There  is  some  level  of  specificity  to  characterize  the  Interaction  (i.e.  ability  to  influence)  between  technologies  through  their  interface. 


Principal  integration  technologies  function  as  stand-alone  units 

Inputs/outputs  for  principal  integration  technologies  are  known,  characterized  and  documented 

Principal  interface  requirements  for  integration  technologies  have  been  defined/drafted 

Principal  interface  requirements  specifications  for  integration  technologies  have  been  defined/drafted 

Principal  interface  risks  for  integration  technologies  have  been  defined/drafted 

Integration  concept/plan  has  been  updated 

Integration  test  concept/plan  has  been  updated 

High-level  Concept  of  Operations  and  principal  use  cases  have  been  updated 
Integration  sequence  approach/schedule  has  been  updated 
Interface  control  plan  has  been  updated 

Integration  and  test  resource  requirements  (facilities,  hardware, software,  surrogates,  etc.)  have  been  updated 
Long  lead  planning/coordination  of  integration  and  test  resources  have  been  initiated 
Integration  &  Test  Team  roles  and  responsibilities  have  been  updated 
Formal  integration  studies  have  been  initiated 

IRL  Criteria 

,  There  is  Compatibility  (i.e.  common  language)  between  technologies  to  orderly  and  efficiently  integrate  and  interact. 


Preliminary  Modeling  &  Simulation  and/or  analytical  studies  have  been  conducted  to  identify  risks  &  assess  compatibility  of  integration  technologies 
Compatibility  risks  and  associated  mitigation  strategies  for  integration  technologies  have  been  defined  (initial  draft) 

Integration  test  requirements  have  been  defined  (initial  draft) 

High-level  system  interface  diagrams  have  been  completed 
Interface  requirements  are  defined  at  the  concept  level 
Inventory  of  external  interfaces  is  completed 
Data  engineering  units  are  identified  and  documented 

Integration  concept  and  other  planning  documents  have  been  modified/updated  based  on  preliminary  analyses 


DoDAF-Described  Models 


SV-1 

SV-1 

CV-3,6 

PV-2 

FFP 

FFP 

CV-1 

FFP 

SV-8,9,1 0 

OV-2 

FFP 

SV-2, 3 

SvcV-lOa 

DoDAF-Described  Models 


SV-1 

SV-2, 5a 

FFP 

SV-2, 6 

SV-1 

FFP 

SV-1 

FFP 

FFP 

FFP 

OV-1 

FFP 

PV-2 

SV-1 

SV-2, 3 

SV-6 

OV-2, 4 

FFP 

DoDAF-Described  Models 


FFP 

FFP 

FFP 

SV-1 

FFP 

SV-3 

3tdV-1,  SV-2 
SV-3, 8 


SV-1 

SV-5a 
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IRL  Criteria 

.  There  is  sufficient  detail  in  the  Quality  and  Assurance  of  the  integration  between  technologies. 


DoDAF-Described  Models 


Quality  Assurance  plan  has  been  completed  and  implemented 
Cross  technology  risks  have  been  fully  identified/characterized 

Modeling  &  Simulation  has  been  used  to  simulate  some  interfaces  between  components 

Formal  system  architecture  development  is  beginning  to  mature 

Overall  system  requirements  for  end  users’  application  are  known/baselined 

Systems  Integration  Laboratory/Software  test-bed  tests  using  available  integration  technologies  have  been  completed  with  favorable  outcomes 
Low  fidelity  technology  “system”  integration  and  engineering  has  been  completed  and  tested  in  a  lab  environment 
Concept  of  Operations,  use  cases  and  Integration  requirements  are  completely  defined 
Analysis  of  internal  interface  requirements  is  completed 
Data  transport  method(s)  and  specifications  have  been  defined 
A  rigorous  requirements  inspection  process  has  been  implemented 
IRL  Criteria 

c  There  is  sufficient  Control  between  technologies  necessary  to  establish,  manage,  and  terminate  the  integration. 


An  Interface  Control  Plan  hasbeen  implemented  (i.e.,  Inter-face  Control  Document  created,  Interface  Control  Working  Group  formed,  etc.) 

Integration  risk  assessments  are  ongoing 

Integration  risk  mitigation  strategies  are  being  implemented  &risks  retired 
System  interface  requirements  specification  has  been  drafted 

External  interfaces  are  well  defined  (e.g.,  source,  data  formats,  structure,  content,  method  of  support,  etc.) 

Functionality  of  integrated  configuration  items  (modules/functions/assemblies)  has  been  successfully  demonstrated  in  a  laboratory/synthetic  environment 
The  Systems  Engineering  Management  Plan  addresses  integration  and  the  associated  interfaces 
Integration  test  metrics  for  end-to-end  testing  have  been  defined 
Integration  technology  data  has  been  successfully  modeled  and  simulation 

IRL  Criteria 

R  The  integrating  technologies  can  Accept,  Translate,  and  Structure  Information  for  its  intended  application. 


Cross  technology  issue  measurement  and  performance  characteristic  validations  completed 

Software  components  (operating  system,  middleware,  applications)  loaded  onto  subassemblies 

Individual  modules  tested  to  verify  that  the  module  components  (functions)  work  together 

Interface  control  process  and  document  have  stabilized 

Integrated  system  demonstrations  have  been  successfully  completed 

Logistics  systems  are  in  place  to  support  Integration 

Test  environment  readiness  assessment  completed  successfully 

Data  transmission  tests  completed  successfully 


FFP 

PV-2 

SV-1 

)V-5a/b,  6a/b 

AV-1 

CV-1 

SV-1, 4 

PV-2 

SV-1 ,4 

PV-3 

FFP 

OV-2 

SV-1 

SV-2 

FFP 

DoDAF-Described  Models 


FFP 

FFP 

FFP 

FFP 

SV-1 

SV-3 

PV-3 

SV-8 

FFP 

SV-7 

SV-5a/5b 

DoDAF-Described  Models 


SV-7 

SV-2 

PV-2 

SV-8 

FFP 

FFP 

PV-2 

FFP 

SV-8, 9 

FFP 

SV-2, 3 
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IRL  Criteria 

7  The  integration  of  technologies  has  been  Verified  and  Validated  with  sufficient  detail  to  be  actionable. 


End-to-end  Functionality  of  Systems  Integration  has  been  successfully  demonstrated 

Each  system/software  interface  tested  individually  under  stressed  and  anomalous  conditions 

Fully  integrated  prototype  demonstrated  in  actual  or  simulated  operational  environment 

Information  control  data  content  verified  in  system 

Interface,  Data,  and  Functional  Verification 

Corrective  actions  planned  and  implemented 

IRL  Criteria 

Actual  integration  completed  and  Mission  Qualified  through  test  and  demonstration,  in  the  system  environment. 


All  integrated  systems  able  to  meet  overall  system  requirements  in  an  operational  environment 

System  interfaces  qualified  and  functioning  correctly  in  an  operational  environment 

Integration  testing  closed  out  with  test  results,  anomalies,  deficiencies,  and  corrective  actions  documented 

Components  are  form,  fit,  and  function  compatible  with  operational  system 

System  is  form,  fit,  and  function  design  for  intended  application  and  operational  environment 

Interface  control  process  has  been  completed/closedout 

Final  architecture  diagrams  have  been  submitted 

Effectiveness  of  corrective  actions  taken  to  closeout  principal  design  requirements  has  been  demonstrated 

Data  transmission  errors  are  known,  characterized  and  recorded 

Data  links  are  being  effectively  managed  and  process  improvements  have  been  initiated 

IRL  Criteria 

q  Integration  is  Mission  Proven  through  successful  mission  operations. 


Fully  integrated  system  has  demonstrated  operational  effectiveness  and  suitability  in  its  intended  or  a  representative  operational  environment 

Interface  failures/failure  rates  have  been  fully  characterized  and  are  consistent  with  user  requirements 

Lifecycle  costs  are  consistent  with  user  requirements  and  life-cycle  cost  improvement  initiatives  have  been  initiated 


DoDAF-Described  Models 


SV-4 

SV-4,8 

SV-8 

SV-6 

SV-1,2,3,4  PV-2 

FFP  SV-9 

DoDAF-Described  Models 


CV-6 

PV-3 

SV-1 ,8 

PV-2, 3 

FFP 

SV-9 

SV-1, 2, 3, 3 

SV-1,2,3,4 

PV-2 

SV-1, 2, 3 

PV-2 

FFP 

FPP 

SV-3 

FFP 

SV-3,8 

DoDAF-Described  Models 


CV-6  SV-8 
FFP  SV-1,7 

FFP 
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